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MULTIPLE INPUT MODES IN OVERLAPPING PHYSICAL 

SPACE 

Background of the Invention 
Field of the Invention 

[0001 ] The present invention relates to input devices for portable electronic de- 
vices, and more particularly to an input device that accommodates multiple input 
modes in the same physical space. 

Description of the Background Art 

[0002] In a virtual keyboard system, a user taps on regions of a surface with his 
or her fingers or with another object such as a stylus, in order to interact with an elec- 
tronic device into which data is to be entered. The system determines when a user's 
fingers or stylus contact a surface having images of keys (" virtual keys"), and further 
determines which fingers contact which virtual keys thereon, so as to provide input to 
a PDA (or other device) as though it were conventional keyboard input The keyboard 
is virtual, in the sense that no physical device need be present on the part of surface 
that the user contacts, henceforth called the work surface. 

[0003] A virtual keyboard can be implemented using, for example, a keyboard 
guide: a piece of paper or other material that unfolds to the size of a typical keyboard, 
with keys printed thereon to guide the user's hands. The physical medium on which 
the keyboard guide is printed is simply an inert surface and has no sensors or me- 
chanical or electronic component The input to the PDA (or other device) does not 
come from the keyboard guide itself, but rather is based on detecting contact of the 
user's fingers with areas on the keyboard guide. Alternatively, a virtual keyboard can 
be implemented without a keyboard guide, so that the movements of a user' s fingers 
on any surface, even a plain desktop, are detected and interpreted as keyboard input. 
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Alternatively, an image of a keyboard may be projected or otherwise drawn on any 
surface (such as a desktop) that is defined as the work surface or active area, so as to 
provide finger placement guidance to the user. Alternatively, a computer screen or 
other display may show a keyboard layout with icons that represent the user's fingers 
superimposed on it. In some applications, nothing is projected or drawn on the sur- 
face. 

[0004] U.S. Patent No. 6,323,942, for ''CMOS Compatible 3-D Image Sensor/ 7 dis- 
closes a three-dimensional imaging system including a two-dimensional array of pixel 
light sensing detectors and dedicated electronics and associated processing circuitry to 
measure distance and velocity data in real time using time-of-flight (TOF) data. 
[0005] The related patent applications referenced above disclose additional data 
input methods, modes, and apparatuses including direct three-dimensional sensing, 
planar range sensors, and vertical triangulation. Such techniques detect user input by 
sensing three-dimensional data to localize the user's fingers as they come in contact 
with a surface. 

[0006] The applications further describe several data input methods, modes, and 
apparatuses for sensing object movements with a sensing device (either 3D, planar, 
vertical triangulation, or otherwise) and interpreting such movements into digital data 
(such as keystrokes). In some of the above applications, techniques are described for 
combining stimuli detected in two or more sensory domains in order to improve per- 
formance and reliability in classifying and interpreting user gestures. These data in- 
put methods are used for entering data into any kind of electronic equipment such as 
mobile devices (e.g. PDA, cell-phone, pen-tablet, computer, etc.) and provide signifi- 
cant benefits over existing methods due to their ease of use, portability, speed of data 
entry, power consumption, weight, and novelty. Many of the described techniques 
are implemented in a virtual keyboard input system in which a user may strike an in- 
ert surface, such as a desktop, on which a keyboard pattern is being projected. 
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[0007] The Senseboard product, offered by Senseboafd Technologies AB of 
Stockholm, Sweden, captures and interprets the motion of a user's fingers in order to 
allow keyboard-like input without a physical keyboard. 

[0008] Conventional sensing devices are typically adapted to detect one particu- 
lar type of input in a particular defined area, such as for example keyboard input. 
However, in many situations if it is desirable to provide two or more input modes. 
For example, most personal computers now provide both mouse and keyboard input 
devices, both of which are often used in quick succession to provide input and to spec- 
ify command and control functions. Conventional sensing devices that operate by de- 
tecting finger motion are unable to perform both input functions in a given detection 
area. 

[0009] MultiTouch products offered by FingerWorks Inc. of Townsend, Delaware 
provide limited capability for receiving typing, mouse, and gesture input in the same 
overlapping area of an input pad. These products use an input detection pad and are 
not able to function on an inert surface such as an ordinary desktop. The overall input 
area is limited to that covered by the active surface, thus reducing the flexibility and 
portability of the device, particularly if it is to be used with personal digital assistants 
(PDAs) or other devices that are usually carried around by users. 

[001 0] What is needed, then, is a system and method for facilitating two or more 
input modes in a sensory input device. What is further needed is a system and 
method for facilitating two or more input modes without requiring separate physical 
space to be designated for each. What is further needed is a system and method for 
facilitating multiple input modes in a small space and on an inert surface such as a 
desktop. What is further needed is a system and method for facilitating multiple input 
modes in a sensory input device without requiring a user to reposition his or her fin- 
gers when switching from one mode to another. What is further needed is a system 
and method for facilitating two or more input modes while preserving the flexibility, 
portability, and other advantages of a sensory input device. 
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Summary of the Invention 

[001 1 ] This invention enables two or more input modes (for instance, keyboard 
and mouse) in an overlapping or coextensive physical space using a sensory input sys- 
tem. The invention is operable on an inert surface such as a desktop. The user moves 
his or her fingers as though interacting with an ordinary input device; the system of 
the invention detects the finger motions and interprets them accordingly. Depending 
on the currently active mode, the invention interprets the finger motions as input ac- 
cording to one of the input modes, and changes its sensory input interpretation tech- 
niques so as to be better adapted to receive and interpret input in the current input 
mode. 

[001 2] In one embo dim ent, the user can switch from one mode to another by 
specifying a mode switch command. In another embodiment, the system of the inven- 
tion automatically detects, from the nature of the user's input, that the input mode 
should be switched, and performs the mode switch accordingly. For example, in an 
embodiment that provides a keyboard mode and a mouse mode, the sensing device of 
the invention detects whether a user appears to be tapping (as one would interact with 
a keyboard) or gliding across the work surface (as one would interact with a mouse). 
Depending on the detected input type, the system of the invention automatically 
switches to the corresponding input mode and interprets the user's finger motions ac- 
cordingly. 

[001 3] In one embodiment, the system of the invention projects an input guide 
onto the work surf ace, so as to help the user in positioning his or her fingers properly. 
In one embodiment, the invention changes the input guide when the input mode 
changes, so as to provide a guide that is appropriate to the current input mode. In an- 
other embodiment, the projected input guide does not change when the mode 
changes. In another embodiment, the system of the invention projects input guides for 
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two or more modes simultaneously. In yet another embodiment, the user is able to 
configure the system regarding whether or not to change the projected input guide 
when the mode changes. 

[001 4] The present invention is able to operate in conjunction with any of the 
various implementations and designs described in the above-referenced related appli- 
cations. For example, the present invention may be implemented in a device that uses 
techniques for combining stimuli in multiple sensory domains as described in U.S. 
Patent Application Serial No. 10/187,032 for "Detecting, Classifying, and Interpreting 
Input Events Based on Stimuli in Multiple Sensory Domains/' 

[001 5] The present invention thus provides many of the advantages of sensory 
input systems that can operate on an inert surface, and provides the further advantage 
of being able to accept input in multiple modes within the same physical space. In 
addition, the present invention is able to change its sensory input interpretation tech- 
niques depending on the current mode, so as to, more accurately capture and interpret 
input in different modes. Furthermore, although the description herein is focused 
primarily on keyboard and mouse input modes, one skilled in the art will recognize 
that the techniques of the present invention can be applied to any sensory input sys- 
tem offering multiple input modes, and that the input modes can correspond to any 
type of physical or virtual input mechanism, including for example: musical instru- 
ments, joysticks, trackballs, jog/ dial controllers, pen-based tablets, and the like. 

Brief Description of the Drawings 

[001 6] Fig. 1 A is a diagram depicting an integrated multiple-mode input device 
displaying a keyboard guide according to one embodiment of the present invention. 
[0017] Fig. IB is a diagram depicting an integrated multiple-mode input device 
displaying a mouse guide according to one embodiment of the present invention. 
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[001 8] Fig. 1C is a diagram depicting an integrated multiple-mode input device 
displaying a combination keyboard/mouse guide according to one embodiment of the 
present invention. 

[001 9] Fig. 2 is block diagram of an embodiment of the present invention. 
[0020] Fig. 3 is an example of a keyboard guide for one embodiment of the pre- 
sent invention. 

[0021 ] Fig. 4 is a flowchart depicting a method for providing multiple input 
modes in an overlapping physical space, according to one embodiment of the present 
invention. 

[0022] Fig. 5 is block diagram depicting dispatching events to appropriate event 
queues, according to one embodiment of the present invention. 
[0023] Fig. 6 is a diagram depicting a stand-alone multiple-mode input device 
displaying a keyboard guide according to one embodiment of the present invention. 
[0024] Fig. 7 is a diagram depicting a stand-alone multiple-mode input device 
displaying a mouse guide according to one embodiment of the present invention. 
[0025] Fig. 8 is an example of a use case illustrating key occlusion. 
[0026] Fig. 9 is another example of a use case illustrating key occlusion. 
[0027] Fig. 10 is another example of a use case illustrating key occlusion. 
[0028] The Figures depict preferred embodiments of the present invention for 
purposes of illustration only. One skilled in the art will readily recognize from the fol- 
lowing discussion that alternative embodiments of the structures and methods illus- 
trated herein may be employed without departing from the principles of the invention 
described herein. 

Detailed Description of a Preferred Embodiment 

[0029] The following description of system components and operation is merely 
exemplary of embodiments of the present invention. One skilled in the art will recog- 
nize that the various designs, implementations, and techniques described herein may 
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be used alone or in any combination, and that many modifications and equivalent ar- 
rangements can be used. Accordingly, the following description is presented for pur- 
poses of illustration, and is not intended to limit the invention to the precise forms dis- 
closed. 

Architecture 

[0030] Referring now to Figs. 1 A through 1C, there is shown a diagram of an in- 
tegrated device 101 that includes apparatus for providing input functionality accord- 
ing to one embodiment of the present invention. Referring also to Figs. 6 and 7, there 
is shown a diagram of a stand-alone device housing 600 that includes apparatus for 
providing input functionality according to one embodiment of the present invention. 
In general, the present invention operates to provide input for any type of device 101, 
which may be a personal digital assistant (PDA), cell phone, or the like. The invention 
may be implemented in an apparatus enclosed within device 101 (as shown in Figs. 1 A 
through 1C) or in a separate housing 600 (as shown in Figs. 6 and 7) that includes ap- 
paratus for sending input signals to a host device. In one embodiment, the present in- 
vention provides mechanisms for implementing data input methods, modes, and ap- 
paratuses including direct three-dimensional sensing, planar range sensors, and verti- 
cal triangulation. Such techniques detect user input by sensing three-dimensional data 
to localize the user' s fingers as they come in contact with surface 204. In one embodi- 
ment, surface 204 is an inert work surface, such as an ordinary desktop. 
[0031] Referring also to Fig. 2, there is shown a block diagram depicting an input 
device according to an embodiment of the present invention. In one embodiment, one 
or two (or more) sensor circuits 106, 108 are provided, each including a sensor 107, 
109. Sensors 107, 109 may be implemented, for example, using charge-coupled device 
(CCD) and/ or complementary metal-oxide semiconductor (CMOS) digital cameras as 
described in U.S. Patent No. 6,323,942, to obtain three-dimensional image information. 
While many of the embodiments shown herein include one sensor 107, one skilled in 
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the art will recognize that any number of sensors can be used, and thus references to 
"a sensor'' are understood to include multiple sensor embodiments. It is beneficial, in 
some embodiments using three-dimensional sensing technology, to position sensors 
107, 109 at the bottom of device 101, so as to more accurately detect finger motions and 
contact with the work surface in the proximity of the bottom of such device. Alterna- 
tively, it may be preferable in some embodiments to position sensors 107, 109 at the 
side and towards the center or above device 101. Such a location may be advantageous 
to provide an improved vantage point relative to the location of the user's fingers on 
the work surface when using two-dimensional sensors such as CCD or CMOS cam- 
eras. 

[0032] Central processing unit (CPU) 104 runs software stored in memory 105 to 
detect input events, and to communicate such events to an application running on 
host device 101. In implementations where the input device of the present invention is 
provided in a separate housing 600 from host device 101 (as shown in Figs. 6 and 7), 
CPU 104 communicates with device 101 via any known port 102 or communication 
interface, such as for example serial cable, Universal Serial Bus (USB) cable, Infrared 
Data Association (irDA) port, Bluetooth port, or the like. Light source 111 iUuminates 
the area of interest on the work surface so that sensors 107, 109 can detect activity. 
[0033] In one embodiment, sensor circuit 106, sensor 107, memory 105, and CPU 
104, as well as circuitries for controlling optional projector 110 and light source 111, 
are integrated into a single CMOS chip or multi-chip module 103, also referred to as a 
sensor subsystem 103. One skilled in the art will recognize that in alternative em- 
bodiments the various components of module 103 may be implemented separately 
from one another. 

[0034] In one embodiment, projector 110 projects an input guide (shown vari- 
ously as 203A, 203B, and 203C in the drawings) onto work surface 204. Guide 203A, 
203B, 203C has a virtual layout that mimics the layout of a physical input device ap- 
propriate to the type of input being detected. For example, in Fig. 1 A and Fig. 6, guide 
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203A has a layout resembling a standard QWERTY keyboard for entering text. In the 
examples of Fig. IB and Fig. 7, mouse input guide 203B is projected, to show the user 
the active area for virtual mouse movement. In the example of Fig. 1C, a combination 
keyboard/ mouse input guide 203C is projected, drawn as a mouse guide overlaying a 
keyboard guide. In one embodiment, whenever a combination keyboard/ mouse in- 
put guide 203C is projected, the mouse guide is projected in a different color than the 
keyboard guide, to further clarify the distinction between the two. Guide 203C indi- 
cates that the user can either type or perform mouse movements, in the same area of 
work surface 204. In one embodiment, as will be described in more detail below, de- 
vice 110 is able to receive mouse input even when keyboard input guide 203A is pro- 
jected, and even when no mouse input guide is projected. In general, one skilled in 
the art will recognize that input guide 203 can take any form appropriate to the cur- 
rently active input mode. 

Multiple Modes 

[0035] The present invention accepts user input in two or more modes. Two or 
more input modes can be implemented in a sensing device by providing separate de- 
tection areas for each input mode. Thus, a mouse area and a keyboard area might be 
defined, possibly having separating sensing apparatus for each. A user wishing to 
provide mouse input moves his or her fingers within the defined mouse area. When 
the user wishes to provide keyboard input, he or she moves his or her fingers within 
the defined keyboard area. In such an implementation, the input mode areas are non- 
overlapping. 

[0036] In a preferred embodiment, the detection areas for the input modes over- 
lap one another, at least in part Such an approach allows each detection area to be 
made larger, and therefore facilitates input within a relatively small desktop area, 
without compromising input detection area size. In addition, such an approach re- 
duces or eliminates the requirement that the user move his or her fingers from one 
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physical area to another when switching between input modes. In one embodiment, 
one detection area wholly includes or is coextensive with another detection area, so 
that the user can keep his or her fingers in the same physical area even when the de- 
vice switches from one input mode to another. 

[0037] For illustrative purposes, in the following description the invention will be 
described in terms of keyboard and mouse input modes. However, one skilled in the 
art will recognize that the techniques of the present invention can be used to imple- 
ment other input modes in any combination. Thus, the invention is not intended to be 
limited to the particular example of a keyboard mode and a mouse mode. 
[0038] When in a keyboard mode, device 101 interprets the user's finger motions 
as keyboard input. Based on sensor 107 detection of the user's finger positions at the 
time the user taps on work surface 204, device 101 determines which keystroke was 
intended. 

[0039] When in a mouse mode, device 101 interprets the user's finger motions as 
though it were input from a pointing device such as a mouse, trackball, trackpad, or 
the like. Based on sensor 107 detection of the user's finger positions and movements 
on work surface 204, device 101 moves an onscreen cursor, activates onscreen objects, 
highlights onscreen text and objects, and performs other activities commonly associ- 
ated with and controlled by pointing devices such as mice. 

[0040] When in other input modes, device 101 interprets the user's finger motions 
in a manner appropriate to the currently active mode. 

[0041] In one embodiment device 101 switches from one mode to another in re- 
sponse to a command from the user. The user may request a mode switch by pressing 
a designated button on device 101, or by performing a predefined gesture or finger 
movement detected by sensor 107 and interpreted by device 101, or by speaking, tap- 
ping, or issuing some other auditory command that is detected and interpreted by de- 
vice 101 according to conventional voice recognition or auditory recognition tech- 
niques. In one embodiment, a number of different mechanisms for commanding a 
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mode switch may be provided, allowing the user to select the mechanism that is most 
convenient at any given time. Recognizing that users often switch rapidly and repeat- 
edly from one mode to another, the present invention makes it very easy and conven- 
ient to perform such switches. 

[0042] Additional examples of mode change mechanisms and commands include, 
without limitation: 

• Pressing a designated virtual key or keys changes into a new mode until 
the same key is pressed again. 

• Pressing a designated virtual key or keys changes into a new mode only 
while the key or keys are depressed. 

• Pressing a specific sequence of virtual keys (e.g. Ctrl-Shift-1) changes 
into a new mode. 

• Specific finger movements change mode. For example, a double tap on 
work surface 204 enters a mode, and triple tap exits a mode. Since a 
sensing system is being used, the finger movements are not limited to 
traditional computing finger movements. New operations such as a 
"pinch," "flick," "wiggle," "scrub," or other type of defined finger 
movement could also change modes. 

[0043] One skilled in the art will recognize that the above list is merely exem- 
plary, and that many other techniques for providing and interpreting mode change 
commands can be used without departing from the essential characteristics of the in- 
vention. In addition, node change commands (and other commands) need not be lim- 
ited to movement along work surface 204. Gestures or other body movements could 
be used to change modes in a 3-dimensional environment. For instance, a thumbs-up 
or thumbs-down gesture could enter and/ or exit a mode. Making a fist could change 
mode, grasping hands together could change mode, and so on. Kicking a leg or shak- 
ing hips could also change mode. 
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[0044] In another embodiment, device 101 automatically switches from one mode 
to another depending on the current context of the user interaction, or under control of 
the host device. For example, a numeric virtual keyboard mode can be activated when 
the context of the user interaction dictates that numeric input is expected. 
[0045] In another embodiment, device 101 automatically switches from one mode 
to another, based on the nature of the detected finger positions and motions of the 
user. For example, if sensor 107 detects that the user has his or her fingers in a typing 
position or is moving his or her fingers in a manner consistent with typing, device 101 
automatically switches to keyboard mode, and interprets finger movements as key- 
strokes. If sensor 107 detects that the user is gliding his fingers along surface 204 in a 
manner consistent with moving a mouse or interacting with a trackpad, device 101 
automatically switches to mouse mode, and interprets finger movements as mouse 
movements. 

[0046] In one embodiment, keyboard and mouse input are distinguished from 
one another by analysis of finger image blob motion. Blob motion representing key- 
board input tends to be essentially vertical, corresponding to the tapping of keys, so 
that when the device detects a quick descent followed by an abrupt stop, it can assume 
keyboard input. By contrast, blob motion representing mouse input tends to have 
small amounts of vertical motion; thus, when the device detects movement parallel to 
the plane of the work surface with minimal vertical movement, it can assume mouse 
input 

[0047] In one embodiment, even if automatic mode switches are provided, device 
101 allows the user to temporarily disable and/or override automatic mode switches. 
Thus, in the event the user's finger movements cause device 101 to make incorrect as- 
sumptions regarding the input mode, or if the user's current activity is specialized or 
limited to one mode, the user is able to control the manner in which his or her actions 
are interpreted. 
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[0048] In one embodiment, the invention provides seamless integration of the 
multiple mode sensory input system with an existing host system such as a personal 
computer or standalone PDA. Referring again to Fig. 2 and also to Fig. 5, CPU 104 
communicates, via port 102, with a device driver 501 on device 101 that interprets the 
incoming events (such as keystrokes, joystick action, or mouse movements) and dis- 
patches those events to an appropriate standard event queue 502-504 for those "vir- 
tual" devices. For instance, the keystrokes are dispatched to key event queue 502, the 
joystick actions to joystick event queue 503, and the mouse events to mouse event 
queue 504. Once the events are in the appropriate queue, device 101 processes the 
events as if they were coming from an actual physical device (such as a physical key- 
board, joystick, or mouse). The invention thereby facilitates "plug-and-play" opera- 
tion in connection with applications already written for the supported device types 
(such as keyboard, joystick, or mouse). In some embodiments, event queue 502-504 is 
implemented as another device driver or is embedded inside another device driver. In 
this case, the invention manipulates the other device drivers to insert the events in the 
driver directly. 

[0049] This device driver system does not limit the functionality to compatibility 
with old applications, however. New applications that can support a more rich or en- 
hanced set of event information are also support by dispatching this more rich set of 
event information directly to them. The invention thereby works with existing legacy 
applications, but also supports new applications with additional functionality. 

Input Guide 

[0050] As described above, and as depicted in Figs. 1 A through 1C, 6, and 7, in 
one embodiment device 101 includes projector 110 for projecting input guide 203 onto 
work surface 204. Input guide 203 is not an essential element of the invention, and in 
some embodiments the user provides input by moving his or her fingers on work sur- 
face 204 without the need for any input guide 203. For example, if the user is control- 
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ling the movement of an onscreen cursor in a mouse mode, the user is generally able 
to provide accurate input without any input guide 203. Accordingly, in some em- 
bodiments, input guide 203 may be switched on or off by the user, by activating a 
command, or input guide 203 may switch on or off automatically depending on which 
input mode is active. In other embodiments, projector 110 may be omitted without 
departing from the essential characteristics of the invention. 

[0051] In embodiments that do include one or more input guides 203, projector 
110 may project a different input guide 203 for each mode. Thus, the particular input 
guide 203 being projected depends on and is appropriate to the current input mode. If 
the currently active mode is a keyboard mode, projector 110 projects a keyboard guide 
203A, as depicted in Figs. 1 A and 6. If the currently active mode is a mouse mode, 
projector 110 projects a mouse guide 203B, as depicted in Figs. IB and 7. Projector 110 
switches from one guide to another in response to input mode changes. 
[0052] Alternatively, in another embodiment projector 110 does not switch guides 
203 automatically. Users may find repeated g^iide-switching distracting. Accordingly, 
in one embodiment, input guide 203 for a first input mode (e.g. keyboard mode) con- 
tinues to be projected even when device 101 switches to a second input mode (e.g. 
mouse mode). In another embodiment, input guide 203 is projected as the superposi- 
tion of input guides 203 for two or more input modes. For example, in Fig. 1C, input 
guide 203C is the superposition of a keyboard input guide and a mouse input guide. 
For clarity, in one embodiment, the two input guides being superimposed are pro- 
jected in different colors, or are otherwise rendered visually distinct from one another. 
[0053] In some embodiments, any or all of the above-described input guide 203 
projection schemes are user-configurable. Thus, for example, device 101 may provide 
configuration options allowing a user to specify whether, and under which conditions, 
a particular type of input guide 203 is projected at any given time. 



14 



WO 03/071411 



PCTYUS03/04530 



[0054] In yet other embodiments, some or all of the guides described above are 
printed on a flat surface (such as a piece of paper), rather than or in addition to being 
projected by projector 110. 

[0055] In yet other embodiments, one or more three-dimensional guides may be 
used. A three-dimensional guide could be implemented as a two-dimensional draw- 
ing of a three-dimensional action that accomplishes a mode-change (or performs some 
other action) or it could, in fact, be a three-dimensional image projected, for example, 
as a hologram. 

* 

Method of Operation 

[0056] Referring now to Fig. 4, there is shown a flowchart depicting a method of 
providing multiple input modes in the same physical space according to one embodi- 
ment of the invention. 

[0057] Device 101 starts 400 in one of the input modes (for example, it may start 
in the keyboard mode). An appropriate input guide 203 is projected 401. The user 
provides input via finger movements on work surface 204 (for example, by typing on 
virtual keys), and device 101 detects and interprets 402 the finger movements using 
techniques described in the above-referenced related patent applications. 
[0058] Device 101 detects 403 a mode-change command, which instructs device 
101 to change to another input mode. As described above, in some embodiments, a 
mode-change command is not needed, and device 101 can change modes automati- 
cally depending on the nature of the detected input. 

[0059] Device 101 then changes input mode 404 so that it now detects movements 
corresponding to the new mode. For example, if the user indicated that he or she is 
about to start performing mouse input, device 101 would change to a mouse input 
mode. In one embodiment, the input modes are implemented using lookup tables de- 
fining each layout and multiple-state machines. 
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[0060] If additional mode changes are specified, steps 402 through 404 are re- 
peated. Otherwise, if the user is done with input 405, the method ends 406. 

Example 

[0061] Referring now to Fig. 3, there is shown an example of a keyboard guide 
203AA, that projector 110 projects onto an inert work surface 204 according to one em- 
bodiment, and that facilitates both a keyboard mode and a mouse mode. 
[0062] Sensors 107, 109 detect the user's finger movements with respect to the vir- 
tual keys shown in the keyboard guide 203 AA. As described in related applications 
cross-referenced above, sensors 107, 109 detect user contact with the virtual keys, and 
device 101 interprets the contact as a keystroke. 

[0063] The user touches cross-hair 301 to switch to a mouse input mode. In one 
embodiment, some indication of mouse input mode is presented, for example by alter- 
ing the color, brightness, or other characteristic of keyboard guide 203 AA. The user 
places a finger on cross-hair 302 and moves his or her finger around to control an on- 
screen cursor. Sensors 107, 109 detect the x-y coordinates of the touch point as the 
user moves his or her finger around. Device 101 interprets these coordinates as mouse 
movement commands, and can further detect and interpret common mouse behaviors 
such as acceleration, clicking, double-clicking, and the like. 

Changing Sensory Input Interpretation Techniques According to Mode 

[0064] In one embodiment, the present invention changes its sensory input inter- 
pretation techniques depending on the current mode, so as to more accurately capture 
and interpret input in different modes. For example, the device may use different sen- 
sory input techniques for detecting mouse input, as opposed to detecting keyboard 
input Mouse input movement differs from keyboard input movement. For example, 
keyboard input tends to include tapping movements that are perpendicular to the 
work surface; mouse input tends to include movement in the plane of the mouse pad. 
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[0065] Events associated with a mouse are different from keyboard events as 
well. While the mouse buttons are processed similarly to keyboard keys, the mouse 
pointer (or other pointing device) has no up or down event; it either rolls on the sur- 
face, or it does not. Lifting a mouse or leaving it in place has approximately the same 
effect. In addition, the main output from a mouse device driver is a sequence of coor- 
dinate pairs (plus button events), while keyboard output generally includes key iden- 
tifiers corresponding to the struck keys. Finally, users often wish to shift the position 
of titie mouse without moving the on-screen cursor, an operation that is typically done 
with a physical mouse by lifting the mouse off of the surface and replacing it in a dif- 
ferent position; this is referred to as "planing/' 

[0066] When interpreting keyboard input, the system of the invention determines 
whether there is contact between a finger and surface 204 either by comparing the 
height of the finger's image blob against a calibration table of expected blob heights, or 
by analyzing the blob's motion. As described above, since keyboard motion is essen- 
tially vertical, contact can be deemed to have occurred when a blob descends quickly 
and then stops abruptly. 

[0067] When interpreting mouse input, as mentioned above, vertical motion 
tends to be small and unpredictable. Thus, rather than detect abrupt changes in blob 
descent, the invention uses blob height to distinguish sliding (moving the virtual 
mouse with the intention of providing input) from planing (adjusting the position of 
the virtual mouse without intending to provide input). Furthermore, as a finger slides 
on the pad, the height of its image blob can change as a result of rather unpredictable 
factors, such as variations in the tilt and orientation of the finger, and in the pressure it 
exerts on the pad, which in turn causes the fingertip to deform. 

[0068] In one embodiment, the system of the invention establishes thresholds that 
are used to determine whether the user intends to glide or plane. If the user's fingers 
are above a certain threshold height, the motion is considered to be a plane, and the 
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onscreen cursor is not moved. If the user's fingers are below the threshold height, the 
motion is considered to be a glide, and the onscreen cursor is moved accordingly. 
[0069] In one embodiment, two thresholds are defined, one for contact and one 
for release. The release threshold is smaller than the contact threshold. When a finger 
first appears, the height of its image blob must exceed the contact threshold before 
contact is declared. Contact is terminated when the blob height goes below the release 
threshold. This hysteresis confers some stability to the discrimination between sliding 
and planing. 

Key Occlusions 

[0070] As described above, in one embodiment the device of the present inven- 
tion changes modes in response to a user's finger movement specifying a mode change 
command. In some situations, the finger movement specifying a mode change com- 
mand may be obscured from the view of sensor 107. For example, it is often the case 
that one of the user's fingers obscures another finger. In one embodiment, upon detec- 
tion of an up-event (keystroke release) the present invention delays the up-event for a 
key for a certain number of frames. If after the predetermined number of frames have 
passed, sensor 107 still detects the finger in the down position, the up-event is dis- 
carded; the assumption is made that the up-event was merely the result of an occlu- 
sion. If the finger is in the up position after the predetermined number of frames has 
passed, the up-event is passed to the application. 

[0071] One skilled in the art will recognize that finger occlusion may take place in 
connection with any finger movements, and is not limited to mode change commands. 
Thus, the following discussion is applicable to any user input, and is not restricted in 
applicability to mode change commands. 

[0072] Referring now to Figs. 8 through 10, there are shown additional examples 
of occlusion. The following description sets forth a technique for handling these oc- 
clusion situations according to one embodiment of the invention. 
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[0073] In Fig. 8, Finger A descends 800, and then Finger B descends behind finger 
A 801. Finger B becomes visible when finger A ascends 802. Finger B then ascends 
803. Since Finger B is occluded by Finger A, sensor 107 does not detect the keypress 
represented by Finger B until Finger A has ascended in 802. The system therefore rec- 
ognizes a down event in 802 rather than in 801. In one embodiment, the system trans- 
mits the down event to host device 101 two frames after Finger A has ascended in 802. 

[0074] In Fig. 9, Finger B descends 900, and then Finger A moves in front of finger 
B 901. Finger B ascends while Finger A stays down 902, and then Finger A ascends 
903. Since sensor 107 cannot detect Finger B's ascent in 902, an up event for Finger B is 
recognized in 903 rather than in 902. 

[0075] In Fig. 10, Finger B descends 1000, and then Finger A moves in front of 
finger B 1001. Finger A ascends while Finger B stays down 1002, and then Finger B as- 
cends 1003. This case should behave exactly as a mechanical keyboard. 

Other Applications 

[0076] The above descriptions sets for the invention as applied to keyboard and 
mouse input modes. One skilled in the art will recognize that other virtual input de- 
vice combinations can be implemented using the present invention. Examples of such 
virtual input device combinations include, without limitation: 

• Keyboard/ gesture-editing facilities. A virtual keyboard is used to type 
characters and then a secondary function is implemented to allow edit- 
ing using finger or hand gestures. 

• Musical instruments. A virtual electronic piano or other instrument 
could be created that allows users to play musical notes as well as differ- 
ent percussion instruments (such as percussion instruments) within the 

same area of work surface 204. 

o 
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• Video Games. Enabling different functions within the same physical 
area can optimize Videogames. For instance, one set of functions causes 
a missile to be fired, while a second set of functions causes a bomb to be 
dropped. 

• Point-of-sale terminals. A keyboard is used to enter data specific to a 
product (for instance, push a button to purchase a product) while a sec- 
ondary function could be used to enter a name to track the order. De- 
pending on context, the input mode would change from being a general 
keyboard to a product-specific one. 

• Context-sensitive input device. Input mode can be changed depending 
on the context of the interaction, or under host system control or instruc- 
tion. For example, a numeric virtual keyboard mode can be activated 
when the context of the user interaction dictates that numeric input is 
expected. 

• Automotive applications. A set of gestures (for example, thumbs-up and 
thumbs-down) may be used to turn the radio up and down in the car. 
Another set of gestures (for example, point upwards and point down- 
wards) may be used to turn the air-conditioning up and down. 

[0077] In the above description, for purposes of explanation, numerous specific 
details are set forth in order to provide a thorough understanding of the invention. It 
will be apparent, however, to one skilled in the art that the invention can be practiced 
without these specific details. In other instances, structures and devices are shown in 
block diagram form in order to avoid obscuring the invention. 

[0078] Reference in the specification to "one embodiment 77 or "an embodiment" 
means that a particular feature, structure, or characteristic described in connection 
with the embodiment is included in at least one embodiment of the invention. The 
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appearances of the phrase "in one embodiment" in various places in the specification 
are not necessarily all referring to the same embodiment. 

[0079] As will be understood by those familiar with the art, the invention may be 
embodied in other specific forms without departing from the spirit or essential charac- 
teristics thereof. For example, the particular architectures depicted above are merely 
exemplary of one implementation of the present invention. The functional elements 
and method steps described above are provided as illustrative examples of one tech- 
nique for implementing the invention; one skilled in the art will recognize that many 
other implementations are possible without departing from the present invention as 
recited in the claims. Likewise, the particular capitalization or naming of the mod- 
ules, protocols, features, attributes, or any other aspect is not mandatory or significant, 
and the mechanisms that implement the invention or its features may have different 
names or formats. In addition, the present invention may be implemented as a 
method, process, user interface, computer program product, system, apparatus, or any 
combination thereof. Accordingly, the disclosure of the present invention is intended 
to be illustrative, but not limiting, of the scope of the invention, which is set forth in 
the following claims. 
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Claims 

[0080] What is daimed is: 

1 1. An input device having for detecting user input in at least two input modes, 

2 comprising: 

3 a sensor, for: 

4 responsive to the input device being in a first input mode, detecting 

5 user movement on or proximate to an inert surface within a 

6 first physical space, and generating a signal responsive to 

7 the detected movement; and 

8 responsive to the input device being in a second input mode, detect- 

9 ing user movement on or proximate to an inert surface 

to within a second physical space, and generating a signal re- 

n sponsive to the detected movement; and 

12 a processor, coupled to the sensor, for: 

13 responsive to the input device being in the first input mode, receiving 

14 and processing the detected signal according to the first in- 

15 put mode; and 

16 responsive to the input device being in the second input mode, re- 

17 ceiving and processing the detected signal according to the 

1 8 second input mode; 

19 wherein at least a portion of the second physical space overlaps at least a por- 

20 tion of the first physical space. 

1 2. The input device of claim 1, wherein the first input mode is a keyboard mode 

2 and the second input mode is a mouse input mode. 
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1 3. The input device of claim 1, wherein the second physical space is coextensive 

2 with the first physical space. 

1 4. The input device of claim 1, further comprising: 

2 a mode controller, coupled to the processor, for switching from one of the input 

3 modes to another of the input modes. 

1 5. The input device of claim 1, wherein: 

2 the processor switches from one of the input modes to another of the input 

3 modes. 

1 6. The input device of claim 1, further comprising: 

2 a mode controller, coupled to the processor, for, responsive to a user command, 

3 switching from one of the input modes to another of the input modes. 

1 7. The input device of claim 1, wherein: 

2 - responsive to a user command, the processor switches from one of the input 

3 modes to another of the input modes. 

1 8. The input device of claim 1, further comprising: 

2 a mode controller, coupled to the sensor, for, responsive to at least one charac- 

3 teristic of the detected finger movement, automatically switching 

4 from one of the input modes to another of the input modes. 

1 9. The input device of claim 1, wherein: 

2 responsive to at least one characteristic of the detected finger movement, the 

3 processor automatically switches from one of the input modes to an- 

4 other of the input modes. 
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1 10. The input device of claim 1, further comprising: 

2 a projector, for projecting an input guide adapted to assist the user in providing 

3 input according to at least one of the input modes. 

1 11. The input device of claim 1, further comprising: 

2 a projector, for: 

3 responsive to the input device being in the first input mode, project- 

4 ing an input guide adapted to assist the user in providing 

5 input according to the first input mode; and 

6 responsive to the input device being in the second input mode, pro- 

7 jecting an input guide adapted to assist the user in provid- 

8 ing input according to the second input mode. 

1 12. The input device of claim 1, further comprising: 

2 a projector, for simultaneously projecting at least two input guides adapted to 

3 assist the user in providing input according to at least two of the in- 

4 put modes. 

1 13. The input device of claim 12, wherein the projector projects each input 

2 guide in a different color. 

1 14. A method for detecting user input in at least two input modes, comprising: 

2 responsive to the input device being in a first input mode: 

3 detecting user movement on or proximate to an inert surface within a 

4 first physical space; 

5 generating a signal responsive to the detected movement; and 

6 processing the detected signal according to a first input mode; and 

7 responsive to the input device being in a second input mode: 
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8 detecting user movement on or proximate to an inert surface within a 

9 second physical space; 

io generating a signal responsive to the detected movement; and 

12 processing the detected signal according to a first input mode; 

12 wherein at least a portion of the second physical space overlaps at least a por- 

13 tion of the first physical space. 

1 15. The method of claim 14, wherein the first input mode is a keyboard mode 

2 and the second input mode is a mouse input mode. 

1 16. The method of claim 14, wherein the second physical space is coextensive 

2 with the first physical space. 

1 17. The method of claim 14, further comprising: 

2 switching from one of the input modes to another of the input modes; and 

3 repeating the detecting, generating, and processing steps. 

1 18. The method of claim 14, further comprising: 

2 receiving a user command indicating a mode switch; 

3 responsive to the user command, switching from one of the input modes to an- 

4 other of the input modes; and 

5 repeating the detecting, generating, and processing steps. 

1 19. The method of claim 14, further comprising: 

2 responsive to at least one characteristic of the detected user movement, auto- 

3 matically switching from one of the input modes to another of the in- 

4 put modes; and 

5 repeating the detecting, generating, and processing steps. 
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20. The method of claim 14, further comprising: 

projecting an input guide adapted to assist the user in providing input accord- 
ing to at least one of the input modes. 

21. The method of claim 14, further comprising: 

responsive to the input device being in the first input mode, projecting an input 

guide adapted to assist the user in providing input according to the 

first input mode; and 
responsive to the input device being in the second input mode, projecting an 

input guide adapted to assist the user in providing input according to 

the second input mode. 

22. The method of claim 14, further comprising: 

simultaneously projecting at least two input guides adapted to assist the user in 
providing input according to at least two of the input modes. 

23. The method of claim 22, wherein simultaneously projecting at least two in- 
put guides comprises projecting each input guide in a different color. 
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